I used to think that improving an AI application mainly meant writing a better prompt. That was mostly true when the task involved one request, one model call, and one response. Once an agent begins working across files, tools, retrieved documents, and a long message history, however, the prompt becomes only one part of the system.
The harder question is no longer just: what should I tell the model? It is what the model should know at this moment, what it can retrieve later, and what should be removed before it becomes a distraction.
That is the problem context engineering tries to solve. It treats context as a limited resource that must be selected, updated, and sometimes discarded throughout an agent's work.
Prompt engineering vs. context engineering
Prompt engineering focuses on how instructions are written and organized. It is especially visible in prompt-focused interactions where the main inputs are the system instructions and the user's request.
Context engineering operates at a broader level. It includes the prompt, but also tool definitions, retrieved files, domain knowledge, examples, memory, message history, and tool results. More importantly, it is iterative: the useful context can change after every action.
The key difference is that prompt engineering shapes instructions, while context engineering manages the model's entire working state over time.
Why more context is not always better
A larger context window allows a model to receive more information, but capacity is not the same as attention. As context grows, relevant facts must compete with stale messages, redundant tool output, and unrelated documents. Model performance tends to degrade gradually rather than collapsing at a single token limit.
Every token consumes some of that budget, but not every token contributes equally to the task. A short architectural decision may matter more than thousands of lines of raw logs. A current test failure may be more useful than the entire conversation that preceded it.
This leads to the guiding principle:
Find the smallest set of high-signal information that gives the model the best chance of producing the desired outcome.
"Smallest" does not mean artificially short. The model still needs enough information to act correctly. It means that every part of the context should have a reason to be there.
The anatomy of effective context
System instructions
System instructions should be clear, direct, and written at the right level of abstraction. Instructions that encode every possible situation as rigid logic become brittle. Instructions that say only "be helpful" or "do the task well" do not provide enough direction.
The useful prompt provides concrete goals, constraints, and decision principles while leaving the model enough flexibility to respond to the actual situation. Distinct sections such as Background, Instructions, Tool guidance, and Output requirements make those responsibilities easier to locate.
Tools
Tools do more than let an agent act. They determine how the agent discovers and introduces new information into its context.
A good tool should have a clear purpose, descriptive parameters, predictable errors, and token-efficient results. Tool overlap is also a context problem: if several tools appear to perform the same task, the model must spend attention deciding which one to use and may make inconsistent choices.
The output matters as much as the interface. Returning the ten relevant lines from a file is often more useful than returning the entire file. A search result with paths and line numbers gives the agent a way to retrieve more only when necessary.
Examples
Examples are useful when they demonstrate representative behavior, not when they attempt to enumerate every edge case. A small set of diverse, canonical examples usually provides a stronger signal than a long list of narrowly written rules.
Examples should clarify judgment: what a good answer looks like, when the agent should stop, when it should ask for help, and how it should respond to failure.
Message history and tool results
Conversation history helps an agent preserve decisions and understand how the task evolved. It also accumulates quickly. Repeated explanations, superseded plans, and old tool results can remain in the context long after they stop being useful.
This makes history an active engineering decision rather than a permanent transcript. The agent may need the conclusion of an earlier investigation without needing every command and raw output that produced it.
Preloaded context vs. just-in-time retrieval
In a typical RAG system, relevant documents are retrieved before the model call and added directly to the context. This works well when the information needed for a task is predictable, limited, and can be identified through a single retrieval step.
Agents can also retrieve context just in time. Instead of loading an entire repository or knowledge base, the initial context can contain lightweight references such as file paths, folder names, stored queries, or links. The agent then uses tools to inspect the information relevant to its current decision.
Consider a coding agent asked to change an unfamiliar project. Loading every source file up front would consume context before the agent even understands the task. A more focused process is:
- Read the project instructions and the user's goal.
- Inspect the file tree and search for relevant symbols.
- Open only the files connected to the requested behavior.
- Make a small change and run the relevant checks.
- Use the test output to decide what context is needed next.
Each observation changes what the agent should inspect next. File names suggest responsibilities, search results reveal dependencies, and test failures point to hidden assumptions. This progressive disclosure lets the agent build understanding without keeping the entire environment in working memory.
Just-in-time retrieval has a cost. Exploration adds latency, and an agent with weak search tools or poor guidance can follow irrelevant paths. In practice, a hybrid strategy is often the most useful: preload stable rules and essential background, then retrieve task-specific details as needed.
Managing context for long-running tasks
In What I Learned About Building Effective Agents, agents are described as systems that repeatedly act, observe, and decide what to do next. Over a long task, that loop creates more history than a model can use effectively. Three techniques help preserve coherence without keeping everything in one context window.
Compaction
Compaction summarizes a long interaction and starts a refreshed context with the information needed to continue. A useful compaction of a coding task might preserve:
- The current goal and completion criteria
- Architectural decisions and their reasons
- Files that were changed
- Tests that passed or failed
- Unresolved problems and the next intended action
It can discard repeated discussion and raw tool output whose conclusions have already been captured. The main risk is compressing too aggressively and losing a detail that becomes important later. Therefore, optimize a compaction process for recall first, then remove information only after observing what is consistently unnecessary.
Structured note-taking
Structured notes move durable state outside the context window. A plan, task list, or project note can record progress across many actions and can be read again after a context reset.
This is different from preserving the full conversation. The note contains the state required to resume: decisions, dependencies, completed work, blockers, and next steps. It acts as external working memory with a much lower context cost.
Sub-agent architectures
Sub-agents isolate work that would otherwise pollute the main context. A lead agent can delegate a focused research or implementation task, let the sub-agent explore with a clean context window, and receive a condensed result.
This pattern is most useful when subtasks are independent enough to investigate separately, and their results can be summarized without losing essential evidence. It is less useful when the work depends on constant shared state or when coordinating the agents costs more than the isolated context saves.
Choose among these techniques based on the task:
- Compaction preserves the flow of one long, evolving interaction.
- Structured notes preserve durable state across milestones or context resets.
- Sub-agents isolate independent, context-heavy exploration.
They can also be combined. A lead agent might keep a structured plan, delegate one research question, and later compact the main conversation while retaining both the plan and the sub-agent's conclusion.
A practical context engineering checklist
Before or during an agent task, I can ask:
- What must the model know before it starts?
- What information can remain as a reference and be retrieved later?
- Which instructions are essential, and which merely repeat examples?
- Do the available tools have distinct and understandable purposes?
- Are tool results returning useful evidence or unnecessary volume?
- Which decisions must survive compaction or a context reset?
- Is part of the task independent enough to isolate in a sub-agent?
- What environmental evidence will tell the agent that it has succeeded?
These questions turn context engineering from a vague goal into a continuous design process. The correct context is not selected once. It is revised as the agent learns more about the task.
My takeaway
Context engineering is not the practice of giving a model as much information as possible. It is the practice of deciding which information deserves the model's attention at each step.
Better prompts still matter, but they cannot compensate for irrelevant documents, ambiguous tools, stale history, or missing task state. Reliable agents need a system that introduces information when it becomes useful, preserves decisions that must survive, and removes details after their value has expired.
The most important shift is to treat context as part of the agent's runtime architecture rather than as a static block of text. The question is not only "What prompt should I write?" It is also "What should the model see now, what can it discover later, and what can it safely forget?"